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We consider the minimum-cut partitioning of a graph into more than two parts using spectral 
methods. While there exist well-established spectral algorithms for this problem that give good 
results, they have traditionally not been well motivated. Rather than being derived from first 
principles by minimizing graph cuts, they are typically presented without direct derivation and then 
proved after the fact to work. In this paper, we take a contrasting first-principles approach in which 
we start with a matrix formulation of the minimum cut problem and then show, via a relaxed 
optimization, how it can be mapped onto a spectral embedding defined by the leading eigenvectors 
of the graph Laplacian. The end result is an algorithm that is similar in spirit to, but different 
in detail from, previous spectral partitioning approaches. In tests of the algorithm we find that it 
outperforms previous approaches on certain particularly difficult partitioning problems. 



I. INTRODUCTION 

Graph partitioning, the division of a graph or network 
into weakly connected subgraphs, is a problem that arises 
in many areas, including finite element analysis, elec- 
tronic circuit design, parallel computing, network data 
analysis and visualization, and others [TJ [5] . In the most 
basic formulation of the problem one is given an undi- 
rected, unweighted graph of n vertices and asked to di- 
vide the vertices into k nonoverlapping groups of given 
sizes, such that the number of edges running between 
groups — the so-called cut size — is minimized. This is 
known to be a computationally hard problem. Even for 
the simplest case where k = 2 it is NP-hard to find the di- 
vision with minimum cut size [3]. Good approximations 
to the minimum can, however, be found using a variety of 
heuristic methods, including local greedy algorithms, ge- 
netic algorithms, tabu search, and multilevel algorithms. 
One particularly elegant and effective approach, which 
is the subject of this paper, is spectral partitioning [4], 
which makes use of the spectral properties of any of sev- 
eral matrix representations of the graph, most commonly 
the graph Laplacian. 

The first spectral partitioning algorithms date back to 
the work of Fiedler in the 1970s [5, 6 J and were aimed at 
solving the graph bisection problem, i.e., the problem of 
partitioning a graph into just two parts. For this prob- 
lem the underlying theory of the spectral method is well 
understood and the algorithms work well. One calcu- 
lates the eigenvector corresponding to the second lowest 
eigenvalue of the graph Laplacian and then divides the 
graph according to the values of the vector elements — the 
complete process is described in Section|TT] More recently, 
attention has turned to the general multiway partitioning 
problem with arbitrary k, which is harder. One simple 
approach is to repeatedly bisect the graph into smaller 
and smaller parts using Fiedler's method or one of its 
variants, but this can give rise to poor solutions in some 
commonly occurring situations. A better approach, and 
the one in widest current use, is to construct the nx (k— 1) 



matrix whose columns are the eigenvectors correspond- 
ing to the second- to fcth-lowest eigenvalues of the Lapla- 
cian. The rows of this matrix then define n vectors of 
k — 1 elements each which are regarded as points in a 
(k — l)-dimensional space. One clusters these points into 
k groups using any of a variety of heuristics, the most 
common being the &;-means method, and the resulting 
clusters define the division of the graph. 

As is common in the computer science literature, the 
presentation of algorithms in this class is not normally 
in the form of an explicit derivation of the algorithm 
(though there is the occasional exception, such as [7]). 
Instead, the algorithm is proposed without justification, 
and justified after the fact by demonstrating that it per- 
forms well on particular partitioning tasks, or that it pro- 
vides certain performance guarantees. Philosophically 
the approach is similar to that of the mathematician who 
demonstrates a result by first conjuring a theorem out of 
thin air and then afterwards proving it to be correct. 

While this approach is perfectly logical and technically 
correct, it does not typically give us much understand- 
ing of an algorithm. For that, it would be better to de- 
rive the algorithm from first principles. The purpose of 
this paper is to give such a derivation for multiway spec- 
tral partitioning. Our main goal in doing so is to gain 
an understanding of why spectral partitioning works, by 
contrast with the traditional presentation which demon- 
strates only that it does work. However, as we will see, 
the algorithm that we derive in the process is not, in fact, 
identical to previous spectral partitioning algorithms and 
in Section |IV| we present results that indicate that our 
algorithm can outperform more conventional approaches 
for certain classes of graphs. 

The previous literature on spectral partitioning is 
extensive — this has been an active area of research, es- 
pecially in the last few years. There is also a large lit- 
erature on the related problem of spectral clustering of 
high-dimensional data sets, which can be mapped onto a 
weighted partitioning problem on a complete graph using 
an affinity matrix. The 1995 paper of Alpert and Yao [7] 
provides an early example of an explicit derivation of 
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a general multiway partitioning algorithm. Their algo- 
rithm is substantially different from the most commonly 
used variants, involving a vector partitioning step, and 
also includes one arbitrary parameter which affects the 
performance of the algorithm but whose optimal value 
is unknown. The algorithms of Shi and Malik [8] and 
Meila and Shi [S] are good examples of the standard 
multiway partitioning using fc-means, although applied 
to the slightly different problem of normalized-cut parti- 
tioning. A number of subsequent papers have analyzed 
these algorithms or variants of them [rUHTS] . Summaries 
are given by von Luxburg [3] , Verma and Meila |14j , and 
Bach and Jordan [TS] , although the discussions are in the 
language of data clustering, not graph partitioning. Per- 
haps the work that comes closest to our own is that of 
Zhang and Jordan |16| . again on data clustering, in which 
partitions are indexed using a set of (fc — l)-dimensional 
"margin vectors," which are oriented using a Procrustes 
technique, as we also do, although other details of their 
approach are different from ours. 

The outline of this paper is as follows. In Section [TT] 
we review the derivation of the standard spectral bisec- 
tion algorithm and then in Section |III| present in de- 
tail the generalization of that derivation to the multi- 
way partitioning problem, leading to a straightforward 
algorithm for multiway partitioning of an arbitrary undi- 
rected graph. In Section |PV| we give example applications 
of this algorithm to a number of test graphs, and demon- 
strate that its performance is similar to, or in some cases 
slightly better than, approaches based on fc-means. In 
Section [V] we give our conclusions and discuss directions 
for future research. 



II. SPECTRAL BISECTION 

The term spectral bisection refers to the special case 
in which we partition a graph into exactly k — 2 parts. 
For this case there is a well-established first-principles 
derivation of the standard partitioning algorithm, which 
we review in this section. Our goal in subsequent sections 
will be to find a generalization to the case of arbitrary fc. 

Suppose we are given an undirected, unweighted graph 
on n vertices, which we will assume to be connected 
(i.e., to have only one component), and we wish to divide 
the vertices into two groups which, for the sake of simplic- 
ity, we will take to be of equal size \n (with n even). We 
define an index variable Si for each vertex i = 1 . . . n such 
that Si = 1 if vertex i belongs to group 1 and s,- = — 1 if 
i belongs to group 2. We note that 



\{StSj + 1) = 



if i,j are in the same group, 
otherwise. 



(1) 



Thus the number of edges within groups is given by 
\ Tlij \ ( s i s j + tyAij , where is an element of the adja- 
cency matrix (having value 1 if there is an edge between i 
and j, and zero otherwise) and the extra factor of ^ com- 



pensates for double counting of vertex pairs in the sum. 
The total number of edges in the entire graph is \ ^2ij -^-ij 
and hence the number of edges between groups — which 
is the cut size R — is given by 

r = | ^2 — \ y^x s i s 3 + 

ij ij 

— I / — SiSj)Aij = j 2^ di — j 2^ SiSjAij, (2) 

ij i ij 

where dj = Y]. A^ is the degree of vertex i. Noting that 
sf = 1 for all i, this equation can be rewritten as 

R 4 ^ ^ di8ij sis j ^ ^ ^ AijSiSj — ^ ^ ^ Lij SiS j , (3) 

ij ij ij 

where Ly = diSij — Aij is the ijih element of the graph 
Laplacian matrix L. Alternatively, we can write R in 
matrix notation as 



R = |s T Ls, 



(4) 



where s is the n-component vector with elements s^. 

Our goal, for a given graph and hence for given L, is 
to minimize the cut size R over possible bisections of the 
graph, represented by s, subject to the constraint that 
the two groups are the same size, which is equivalent to 
saying that J^. Si = or 



0, 



(5) 



where 1 = (1,1,1,...). Unfortunately, as mentioned in 
the introduction, this is a hard computational problem. 
But one can in many cases find reasonably good solutions 
in polynomial time by using a relaxation method. We 
generalize the discrete variables Si — ±1 to continuous 
real variables Xi and solve the relaxed minimization with 
respect to the vector x = {x\, 22, ■ ■ ■) of 



R x = |x T Lx, 



subject to the constraint 



l T x = 0, 



(G) 



(7) 



which is the equivalent of Eq. ([5]). One must however 
also apply an additional constraint to prevent x from 
becoming zero, which is normally taken to have the form 



T 

x x 



(8) 



Choices of x satisfying this second constraint include all 
allowed values of the original unrelaxed vector s, since 
s T s = J2i s f — J2i 1 = n ) but also include many other 
values in addition. Geometrically, one can think of s as 
defining a point in an rt-dimcnsional space, with the al- 
lowed values Si = ±1 restricting the point to fall at one 
of the corners of a hypercube. The value of x falls on 
the circumscribing hypersphere, since x T x = J^. xf = n 
implies that x has constant length *Jn. The hypersphere 
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FIG. 1: The possible values of the vector s lie at the corners 
of a hypercube in n-dimensional space, while the vector x can 
lie at any point on the circumscribing hypersphere. 



coincides with the values of s at the corners of the hyper- 
cube, but includes other values in between as well — see 
Fig.[T] 

The relaxed problem is straightforward to solve by dif- 
ferentiation. We enforce the two conditions Q and |8]) 
with Lagrange multipliers A and /i so that 



d 

dx k 



0. 



(9) 



Performing the derivatives we find that 2^ • L^x^ — 
2Xxk — \i = or in matrix notation Lx = Ax + Ajiil. 
Multiplying on the left by 1 we get l T Lx = Al T x+ |n/Lt, 
and employing Eq. ^ and noting that 1 is an eigenvalue 
of L with eigenvalue zero, we find that /i = 0. Thus we 
end up with 



Lx = A> 



(10) 



In other words x is an eigenvector of the graph Laplacian 
satisfying the two conditions ([7 1 and 

Our solution is completed by noting that the cut 
size R x within the relaxed approximation, evaluated at 



the solution of (10 1, is 



R x = |x T Lx = |Ax T x 



nX 



(11) 



This is minimized by choosing A as small as possible, in 
other words by choosing x to be the eigenvector corre- 
sponding to the lowest possible eigenvalue. The lowest 
eigenvalue of L is always zero, with corresponding eigen- 
vector proportional to 1, but we cannot choose this eigen- 
vector because it is forbidden by the condition l T x = 0, 
which requires that the solution vector x be orthogonal 
to 1. (This is equivalent to saying that we're not allow 
to put all vertices in the same one group, which would 
certainly ensure a small cut size, but wouldn't give a bi- 
section of the graph.) Our next best choice is to choose 
x proportional to the eigenvector for the second-lowest 
eigenvalue, the so-called Fiedler vector. 
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FIG. 2: Left: a small, computer-generated graph with 
two equally sized groups of vertices. Right: the elements 
of the Fiedler vector — the second eigenvector of the graph 
Laplacian — plotted on an arbitrary scale. A division of the 
vertices into two groups according to the signs of these ele- 
ments (the dashed line indicates zero) recovers the groups. 



This solves the relaxed optimization problem exactly. 
The final step in the process is to "unrelax" back to the 
original variables Sj = ±1, which we do by rounding the 
Xi to the nearest value ±1, which means that positive val- 
ues of rounded to +1 and negative values to — 1. 
Thus our final algorithm is a straightforward one: we 
calculate the eigenvector of the graph Laplacian corre- 
sponding to the second-lowest eigenvalue and then di- 
vide the vertices into two groups according to the signs 
of the elements of this vector. Although the solution 
of the relaxed optimization is exact, the unrelaxation 
process is only an approximation — there is no guaran- 
tee that rounding to ±1 gives the correct optimum for 
the unrelaxed problem — and hence the overall algorithm 
only gives an approximate solution to the partitioning 
problem. In practice, however, it appears to work well. 
Figure [2] shows an example. 

As we have described it, the algorithm above also does 
not guarantee that the two final groups of vertices will be 
equal in size. The relaxed optimization guarantees that 
Y^,i Xi = because of Eq. ([7]), but we are not guaranteed 
that ^2* Si — after the rounding procedure. Normally 
J2i Si will be close to zero, and hence the groups will be 
of nearly equal sizes, but there can be some imbalance. 
In typical usage, however, this is not a problem. In many 
applications one is willing to put up with a small imbal- 
ance anyway, but if one is not then a post-processing step 
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k = 2 



Jc = 3 



lc = 4 



FIG. 3: As group labels we use vectors Wi pointing from the 
center to the corners of a regular (k — l)-dimensional simplex. 
For k — 2 the simplex consists of just two points on a line, 
consistent with the indices Sj = ±1 used in Section [TTJ For 
k — 3 the simplex is an equilateral triangle; for k — 4 it is a 
regular tetrahedron. In higher dimensions it takes the form 
of the appropriate generalization of a tetrahedron to four or 
more dimensions, which would be difficult to draw on this 
two-dimensional page. 



can be performed that moves a small number of vertices 
between groups in order to restore balance. 



the rows of a fc x [k— 1) matrix W, in which case Eq. ( 12 ) 
is equivalent to WW T = 1— 11 T /k, where I is the iden- 
tity and 1 = (1,1,1,...) as previously. The matrix also 
satisfies an orthogonality condition on its columns of the 
form 



W W = I. 



(13) 



We make use of these simplex vectors in the partition- 
ing problem by assigning one vector to represent each of 
the groups. All assignments are equivalent, up to rota- 
tions and reflections, and any choice will work equally 
well. We label each vertex i with a vector variable s, 



equal to the simplex vector for its group. Then Eq. ( 12 ) 
implies that 



S,- S-i 



1 

k 



if i,j are in the same group, 
otherwise, 



(14) 



which is the equivalent of Eq. 
the cut size follows through as 



TJ, and the derivation of 
aefore, giving 



R 



2 / i 



(diSi 



A,. 



2 L %3 



T 
S,- S i 



(15) 



III. GENERALIZATION TO MORE THAN TWO 
GROUPS 

Our primary purpose in this paper is to give a gen- 
eralization of the derivation of the previous section for 
spectral partitioning into more than two groups. As we 
will see, a true generalization leads to an algorithm that 
differs in significant ways from previous multiway parti- 
tioning algorithms. 



A. Cut size for multiway partitioning 

To generalize the spectral bisection algorithm to the 
case of more than two groups we need first to find an 
appropriate generalization of the quantities Si used in 
Section [TT] to denote membership of the different com- 
munities. For a partitioning into k groups, we propose 
using vectors that point to the k vertices of a (fc — 1)- 
dimensional regular simplex centered on the origin. For 
k — 3, for example, the three vectors would point to the 
corners of an equilateral triangle; for fc = 4 the vectors 
would point to the corners of a regular tetrahedron, and 
so forth — see Fig. [3] 

Such simplex vectors are not orthogonal. Rather they 
satisfy a relation of the form 



1 



W W = % - 



(12) 



Note that the individual vectors are normalized so that 
w^w i = 1 — 1/fc. (One could normalize them to have unit 
length, but subsequent formulas work out less neatly that 
way.) One can also consider the k simplex vectors to be 



where is once again an element of the graph Laplacian 
matrix L. Alternatively, we can introduce an n x (fc — 1) 
matrix S whose ith row is equal to and write the cut 
size in matrix notation as 



R = \ Tr(S T LS) 



(16) 



Since the vertices of the regular simplex are all equiva- 
lent, the vectors pointing to them necessarily sum to zero 
J2i w i = 0. Assuming once again that we want to divide 
the graph into equally sized groups, we can enforce this 
equality by requiring that J2i s i — 0> or equivalently 



I 1 S = 0. 



(17) 



For equally sized groups Eq. ( 13 ) implies that the ma- 
trix S also automatically satisfies the condition 



(18) 



which is the equivalent of the condition that sf — n 
in the two-group case. 



B. Minimization of the cut size 



Our goal is to minimize the cut size ( 16 1 subject to the 
constraint (17 1. Once again this is a hard computational 



problem, but, by analogy with the two-group case, we can 
render it tractable by relaxing the requirement that each 
row of S be equal to one of the discrete simplex vectors, 
solving this relaxed problem exactly, then rounding to 
the simplex vectors again to get an approximate solution 
to the original unrelaxed problem. 
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Specifically, we replace S with a matrix X of 
continuous- valued elements to give a relaxed cut size 



R x = \ Tr(X T LX) 



(19) 



where the elements of X will be allowed to take any real 
value subject to the constraint 



1 T X = 0, 



(20) 



equivalent to Eq. (17). Again, however, we also need an 
additional constraint, equivalent to Eq. to prevent all 
elements of X from becoming zero (which would certainly 
minimize R x , but would not give a useful partition of the 
graph). The natural choice for this additional constraint 



is the generalization of ( 18 ) 



X T X = ^I. 

k 



As in the two-group case, choices of X satisfying this con- 
dition necessarily include as a subset the original simplex 
vectors that satisfy (18), but also include many other 



choices as well. Between them, the two conditions (20) 



and (21 ) imply that the columns of X should be identi- 



cally normalized, orthogonal to one another, and orthog- 
onal to the vector 1. As we now show, the correct choice 
that satisfies all of these conditions and minimizes R x 
is to make the columns proportional to the eigenvectors 
of the graph Laplacian corresponding to the second- to 
/cth-lowest eigenvalues. 



The relaxed cut size ( 19 ) can be minimized, as before, 



by differentiating, applying the conditions (20) and (21) 
with Lagrange multipliers: 



d 



dX. 



kl 



(22) 



so that 2J2j L kjXji - 2Y, m X km X m i - fi t = 0, or in 
matrix notation LX = XA + ^lfi T , where A is a (k — 
1) x (k — 1) symmetric matrix of Lagrange multipliers 
and fi is a (k — l)-dimensional vector. As before, we can 
multiply on the left by 1 T to show that /i = 0, and hence 
we find that 



LX = XA. 



(23) 



For a matrix X satisfying this equation the relaxed cut 
size R x , Eq. ( 19 1, is 



Ra 



\ Tr(X T LX) = \ Tr(X T XA) 



2 — >- 2 — - 2k — (24) 



TrA, 



where we have used Eq. (21 ) 



However, neither the remaining Lagrange multiplier 
matrix A nor the matrix X itself is completely deter- 



mined by Eq. (23) and the conditions (20) and (21), and 



for good reason. Suppose we have a correct solution for 
X and a corresponding A such that Eqs. (20) and (21) 



are satisfied and ( 19 1 is minimized. Then consider an- 
other matrix X' = XQ where Q is any {k — 1) X (k — 1) 
orthogonal matrix. In that case X' also satisfies (20) 
and (plj): 



1 T X' = 1 T XQ = 0, 

X T X = Q T X T XQ = ^Q T Q = "i, 

k k 



(25) 
(26) 



and substituting X = X'Q T into Eq. (23) gives 
LX'Q T = X'Q T A, or equivalently 



LX' = X'Q T AQ. 



(27) 



(21) In other words, if a solution X exists for Eq. (23), then 



any rotation X' is also a solution of the same equation 
but with transformed Lagrange multipliers A' = Q T AQ. 
But A' will have the same trace as A, since the trace 
is invariant under an orthogonal transformation of this 
kind, and hence the value of R x , Eq. (24), is unchanged, 
meaning this new solution also minimizes R x . Thus any 
rotation of a minimum of R x is also a minimum. This is 
natural — only the geometry of the vertices relative to one 
another matters and not their absolute positions. The 
same is also true of the original simplex vectors, which 
can be freely rotated without affecting the cut size. 

What this means in practice is that we are at liberty to 
choose any rotation of the solution we prefer, whichever is 
most convenient. All will give equally good values for the 
relaxed cut size R x . But when we "unrelax" the solution 
again to get back to the original simplex vectors we must 
bear in mind that different rotations do not necessarily 
give the same unrelaxed solution — some rotations may 
be better than others. This means that the unrelaxation 
process may require an additional rotation to find the 
best solution, an extra step in the algorithm that we will 
perform using a polar decomposition. 

For our purposes the most convenient rotation of the 
solution is the one for which Q T AQ is diagonal. Such a 
rotation always exists since A is real and symmetric. For 
this choice Eq. ([27]) takes the form of Eq. ( 23 ) again but 



with A diagonal, which means that each column of X is 
an eigenvector of the graph Laplacian with the diagonal 
elements of A giving the eigenvalues. Then the conditions 
(20) and (21) tell us that the eigenvectors must be dis- 



tinct (because they are orthogonal), identically normal- 
ized, and orthogonal to the vector 1. Equation (24) tells 



us that to minimize R x we should choose the eigenvalues 
as small as possible and, given that we are forbidden by 



Eq. (20) from choosing the lowest (zero) eigenvalue, be- 
cause its eigenvector is the vector 1, our best choice is 
to choose the columns of X to be the eigenvectors cor- 
responding to the second- to fcth-lowest eigenvalues of 
the Laplacian. Which column is which is arbitrary and 
unimportant, again corresponding only to a rotation — 
any choice will work equally well. 
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C. Reversing the relaxation 

The developments of the previous section give us an 
exact solution to the relaxed optimization problem. The 
final step in the algorithm is the reversal of the relax- 
ation process to recover an approximation to the ideal 
partitioning of the graph. In principle this is, as with the 
two-group case, a straightforward operation — we round 
each row of the matrix X to the nearest simplex vector. 
More specifically, we write the ith row Xj of the matrix 
as a linear combination of the simplex vectors Wj : 



(28) 



(This looks like a standard orthogonal decomposition, 
but it's not because the simplex vectors are not orthogo- 
nal. Nonetheless it is straightforward to prove that such 
a decomposition always exists when the vectors Wj point 
to the corners of a regular simplex.) In rounding the 
points Xj — > MVj we choose that simplex vector Wj whose 
contribution, equal to the inner product wjx^, is largest 
in Eq. (28). An equivalent statement is that we simply 



round the points to the nearest corner of the simplex, 
defined in terms of ordinary Euclidean distance in the 
(k — l)-dimensional space. As in the two-group case, this 
is merely a heuristic — there is no guarantee that it will 
give an optimal partitioning, although as we will see it 
appears to work well in practice. 

This, however, is not the whole story. As we mentioned 
in the previous section, there is a complication: both the 
rows of X and the simplex vectors are only fixed up to a 
rotation, so an additional rotation may be required before 
rounding to find the best solution. We can rotate either 
the rows of X or the simplex vectors. We recommend 
rotating the simplex vectors, since there are only k of 
them, whereas there are n rows of the matrix X. 

The situation is depicted in Fig. [4] for the case k = 3. 
The rows Xj are two-dimensional vectors in this case and 
form a scatter of points in the plane of the plot as shown. 
The points do indeed approximate reasonably well to the 
corners of a simplex (an equilateral triangle in this case) , 
so in principle we should be able to round them off and 
get a good solution to the partitioning problem. But we 
don't know a priori what the correct orientation of the 
simplex is, and in this case our first guess, shown as the 
Y shape in the figure, is off and a rotation is required. 
Since our goal is to round the Xj to simplex vectors Wj 
such that the inner products wjx^ are large, a sensible 
criterion for selecting the best rotation (there are others) 
is to maximize the sum of the inner products for all ver- 
tices. 

Given an assignment of vertices to groups, we can write 
down the matrix S of (unrotated) simplex vectors, and 
the rotated vectors take the form SQ, where Q is a (k — 
1) X (k — 1) orthogonal matrix as before. Then the sum 
of the inner products of the rotated vectors with the Xj 
is Tr(Q T S T X) and the task of finding the best rotation 
is equivalent to maximizing this trace over all possible 




FIG. 4: The points in this plot represent the elements of the 
second and third eigenvectors of the Laplacian for a small 
graph of about a thousand vertices. The particular graph 
used here was, for the purposes of illustration, deliberately 
created with three communities in it, using a simple planted 
partition model ]T? in which edges are placed between ver- 
tices independently at random, but with higher probability 
between vertices in the same group than between those in 
different groups. As the figure shows, there is a clear clus- 
tering of the resulting points into three groups, which fall, 
roughly speaking, at the corners of a two-dimensional regular 
simplex, i.e., an equilateral triangle. To determine the divi- 
sion of the graph into groups, we need to round these points 
to the nearest simplex vector (the three ends of the Y shape) , 
but the simplex must first be rotated to match the orientation 
of the points. 



orthogonal matrices Q. This maximization is a standard 
problem in so-called Procrustes analysis [18]. It can be 
solved by performing a singular value decomposition of 
the matrix S T X: 



S T X = USV T , 



(29) 



where S is the diagonal matrix of singular values and U 
and V are orthogonal matrices. Then 

Tr(Q T S T X) = Tr(Q T USV T ) = Tr(V T Q T U£) 

<TrS, (30) 

the inequality following because V T Q T U, being a prod- 
uct of orthogonal matrices, is also itself orthogonal, and 
all elements of an orthogonal matrix are less than or equal 
to 1. It is now trivial to see that the exact equality — 
which is, by definition, the maximum of Tr(Q T S T X) 



with respect to Q 
equivalently when 



-is achieved when Q = VU or 



Q = UV 3 



(31) 



The product UV T is the orthogonal part of the polar 
decomposition of S T X. Calculating it in practice in- 
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volves calculating first the singular value decomposition, 
Eq. (29), and then discarding the diagonal matrix S. 
Note that S T X is only a (k — 1) X (k — 1) matrix (not an 
nxn matrix) , and hence its singular value decomposition 
can be calculated rapidly provided k is small, in 0(fc 3 ) 
time. 

These developments assume that we know the assign- 
ment of the vertices to the groups. In practice, however, 
we don't. (If we did, we wouldn't need to partition the 
graph in the first place.) So in the algorithm we pro- 
pose we start with a random guess at the orientation of 
the simplex. We round the rows of X to the simplex 
vectors to determine group memberships and then rotate 
the simplex vectors to fit the resulting groups according 
to Eq. (31 ). We repeat this procedure until the groups no 
longer change. In a clear-cut case like that of Fig. [4j only 
one or two iterations would be needed for convergence, 
but in more ambiguous cases we have found that as many 
as half a dozen or more iterations may be necessary. 

As in the two-group case, we are not guaranteed that 
the groups found using this method will be exactly equal 
in size. The relaxed optimization must satisfy Eq. (20), 
but the corresponding condition, Eq. (17), for the un- 
relaxed division of the graph is typically only satisfied 
approximately and hence the groups will only be approx- 
imately equal in size. As in the two-group case, however, 
this is typically not a problem. Often we arc content with 
nearly equal groups, but if not the groups can be balanced 
using a post-processing step. For example, the division 
into equally sized groups that maximizes the sum of the 
inner products wjx^ can be calculated exactly in polyno- 
mial time using the so-called Hungarian algorithm [T§] . 
or approximately using a variety of vertex moving heuris- 
tics. 



D. Summary of the algorithm and running time 

Although the derivation of the previous sections is 
moderately lengthy, the final algorithm is straightfor- 
ward. In summary the algorithm for partitioning a given 
graph into k groups is as follows. 

1. Compute the n X (k — 1) matrix X whose columns 
consist of the eigenvectors of the graph Laplacian 
corresponding to the second- to fcth-lowest eigen- 
values. 

2. Choose at random an initial orientation of the (k — 
l)-dimensional regular simplex and construct the 
k x (fc— 1) matrix W whose rows are the vectors Wj 
that point to the corners of the simplex. 

3. Compute the inner products wjxi for all vertices i 
and groups j, then assign each vertex i to the 
group j for which the product is largest. 

4. Using the resulting assignment of vertices to 
groups, compute the matrix S whose ith row is the 
simplex vector Wj for the group j to which vertex i 



belongs, then compute the matrix of rotated sim- 
plex vectors W = WUV T , where USV T is the 
singular value decomposition of the (fc — 1) x (fc — 1) 
matrix S T X. 

5. Replace W by W and repeat from step 3 until the 
group memberships no longer change. 

Because the initial orientation of the simplex is random, 
the results of the algorithm are not deterministic, and 
hence, as with other randomized algorithms, it may be 
beneficial to run the calculation several times with dif- 
ferent initial orientations and among the divisions of the 
graph so generated choose the one that has the smallest 
cut size. A similar procedure is typically used, for exam- 
ple, for partitioning with fc-means, which also employs a 
random initial condition. (Note that the computation of 
the eigenvectors — step 1 above — does not have to be re- 
peated; only the randomized part of the algorithm, from 
step 2 onward, need be repeated.) 

In practice the algorithm runs quickly. Most often 
we are interested in sparse graphs in which the num- 
ber of edges is proportional to the number of vertices, 
so that the mean degree of a vertex tends to a con- 
stant as the graph becomes large. In this situation 
the eigenvectors of the Laplacian can be calculated us- 
ing sparse methods such as the Lanczos algorithm in 
time 0(k 2 n). The other steps of the algorithm all also 
take time 0(fc 2 n), except for the singular value decompo- 
sition, which takes time 0(fc 3 ), and hence (since fc < n) 
one iteration of the algorithm has leading-order worst- 
case running time 0(fc 2 n). We do not have a rigorous 
understanding of how the number of iterations will scale 
with n and fc but the number appears in practice to be 
small — less than about ten iterations for the graphs we 
study — and to increase only slowly, if at all, with graph 
size. We conjecture, therefore, that the running time of 
the algorithm scales approximately as fc 2 n, making it well 
suited for large graphs. 



E. Weighted graphs and data clustering 

The methods described in the previous sections can 
be extended in a straightforward manner to weighted 
graphs — graphs with edges of varying strength repre- 
sented by varying elements in the adjacency matrix. For 
such graphs the goal of partitioning is to divide the ver- 
tices into groups such that the sum of the weights of the 
edges running between groups is minimized. To achieve 
this we generalize the degree di of vertex i in the obvious 
fashion di = J2j Aij an d the elements of the Laplacian 
accordingly Lij = diSij — Aij. Then the cut size once 



again satisfies Eq. (16), and the rest of the algorithm fol- 



lows as before. We have not experimented extensively 
with applications to weighted graphs, but in preliminary 
tests the results look promising. 

One can also apply our methods to the problem of data 
clustering, the grouping of points within a multidimen- 



sional data space into clusters of similar values [JJ Q3] . 
One standard approach to this problem makes use of an 
affinity matrix. Suppose one has a set of n points repre- 
sented by vectors in a d-dimcnsional data space. One 
then defines the affinity matrix A to have elements 

i 1J=e - |r -- r ^ /2ff2 , (32) 

where a is a free parameter chosen by the user. If a 
is roughly of order the distance between intra-cluster 
points, then Aij will approximately take the form of the 
adjacency matrix of a weighted graph in which vertices 
are connected by strong edges if the corresponding data 
points are near neighbors in the data space. (For values 
of a much larger or smaller than this clustering methods 
based on the affinity matrix will not work well, so some 
care in choosing a is necessary to get good results. Auto- 
mated methods have been proposed for choosing a good 
value [TO].) 

Given the affinity matrix, we can now apply the 
method described above for weighted graphs to this ma- 
trix and derive a clustering of the data points. We will 
not pursue this idea further in the present paper, but 
in preliminary experiments on standard benchmark data 
sets we have found that the algorithm gives results com- 
parable with, and in some cases better than, other spec- 
tral clustering methods. 




FIG. 5: Division of a structural engineering mesh network 
of 15 606 vertices into four parts — represented by the four 
colors — using the algorithm described in this paper. The com- 
plete graph has 45 878 edges; this division cuts just 362 of 
them, less than 1%. Graph data courtesy of the University of 
Florida Sparse Matrix Collection. 



IV. RESULTS 

Our primary purpose in this paper is to provide a first- 
principles derivation of a spectral partitioning algorithm. 
However, given that the algorithm we have derived dif- 
fers from standard algorithms, it is also interesting to 
examine how well it performs in practice. In this sec- 
tion we give a number of example applications of the 
algorithm to graphs from a variety of sources. Our tests 
do not amount to an exhaustive characterization of per- 
formance, but they do give a good idea of the basic pic- 
ture. Overall, we find that the algorithm has performance 
comparable to that of other spectral algorithms based on 
Laplacian eigenvectors, but there exist classes of graphs 
for which our algorithm does measurably better. In par- 
ticular, the algorithm appears to perform better than 
some competitors in cases where the partitioning task is 
particularly difficult. 

As a first example, Fig. [5] shows the result of applying 
our algorithm to a graph from the widely used University 
of Florida Sparse Matrix Collection. This graph is a two- 
dimensional mesh network drawn from a NASA struc- 
tural engineering computation, and is typical of finite- 
element meshes used in such calculations (which are a 
primary application of partitioning methods). Figure [5] 
shows a split of the graph into four parts. The split 
is closely similar to that found by conventional spectral 
partitioning using fc-means, indicating that our algorithm 
has comparable performance on this application to stan- 
dard methods in current use. 



Figure [6] shows an application to a graph representing 
a power grid, specifically the Western States Power Grid, 
which is the network of high-voltage electricity trans- 
mission lines that serves the western part of the United 
States 20J . The figure shows the result of splitting the 
graph into four parts and the split is an intuitively sensi- 
ble one and again comparable to that found using more 
traditional methods. 

There are, however, some graphs for which our method 
gives results that are measurably, if only modestly, better 
than those given by previous methods. As a controlled 
test of the performance of the algorithm we have applied 
it to artificial graphs generated using a planted partition 
model 17 (also called a stochastic block model in the sta- 
tistical literature [H]). In this model one creates graphs 
with known partitions by dividing a specified number of 
vertices into groups and then placing edges within and 
between those groups independently with given proba- 
bilities. In our tests we generated graphs of 900 ver- 
tices with three equally sized groups of 300 vertices each. 
Edges were placed between vertices with two different 
probabilities, one for vertices in the same group and one 
for vertices in different groups, chosen so that the aver- 
age degree of a vertex remained constant at 30. We then 
varied the ratio of in-group to between-group connections 
to test the performance of the algorithm. 

Figure [7] shows the results of applying both our algo- 
rithm and the standard fc-means spectral algorithm to 
a large set of graphs generated using this method. We 
compared the divisions found by each algorithm to the 
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FIG. 6: Division into four parts of a graph representing the 
Western States Power Grid of the United States. The com- 
plete graph contains 4941 vertices and 6594 edges, of which 
33 are cut in this division. Graph data courtesy of Duncan 
Watts. 



known correct divisions and calculated the fraction of 
vertices classified into the correct groups as a function of 
the average number c ou t of edges from a vertex to vertices 
in other groups. For small values of c ou t the groups in the 
graph should be clear and we expect any algorithm to do 
a good job of finding the best cut. But as c out is increased 
the problem becomes more challenging and the fraction 
of correct vertices declines for both algorithms, eventu- 
ally approaching the value |, denoted by the horizontal 
dashed line in the figure, at which classification is no bet- 
ter than chance — we would expect a random division of 
vertices into groups to get a third of the vertices right 
just by luck. Just before we reach this point, however, 
in the regime corresponding to the hardest partitioning 
problems that can still be solved by these algorithms, 
we see that our algorithm does measurably better than 
fc-means, and the difference, while small, is statistically 
significant given the errors on the measurements. In this 
regime, therefore, in the context of this particular test, 
it appears that our algorithm returns the better results. 

To be fair, we should also point out that there are 
some cases in which the fc-means algorithm outperforms 
the algorithm of this paper. In particular, we find 
that for graphs generated using the planted partition 
model, again with three groups, but where the between- 
group connections are asymmetric and one pair of groups 
is more weakly connected than the other two pairs, 
the fc-means algorithm does better in certain parameter 
regimes. The explanation for this phenomenon appears 
to be that our algorithm has difficulty finding the best 
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FIG. 7: Fraction of vertices classified into the correct groups 
by a standard spectral algorithm based on fc-means and by the 
algorithm described in this paper, when applied to graphs 
artificially generated using a planted partition model. The 
dashed horizontal line represents the point at which the algo- 
rithms do no better than chance. For both algorithms we use 
a random initial condition as described in the text and the 
results reported are the best of five runs. 



orientation of the simplex to perform the partitioning. 
It is possible that one could achieve better results us- 
ing a different method for finding the orientation. The 
fc-means partitioning algorithm, which does not use an 
orientation step, has no corresponding issues. 



V. CONCLUSIONS 

In this paper, we have derived a spectral partitioning 
algorithm from first principles, as a relaxation approx- 
imation to a well-defined minimum-cut problem. This 
contrasts with more traditional presentations in which 
an algorithm is proposed ex nihilo and then proved after 
the fact to give good results. While both approaches have 
merit, ours offers an alternative viewpoint that helps ex- 
plain why spectral algorithms work — because the spec- 
tral algorithm is, in a specific sense, an approximation 
to the problem of minimizing the cut size over balanced 
divisions of the graph. 

Our approach not only offers a new derivation, how- 
ever; the end product, the algorithm itself, is also dif- 
ferent from previous algorithms, involving an additional 
rotation step that is performed using a polar decompo- 
sition. In practice, the algorithm appears to give results 
that are comparable with those of previous algorithms 
and in some cases measurably, if only modestly, better. 
The algorithm is also efficient. For graphs of n vertices 
divided into fc groups, the running time is 0(fc 2 n) for 
a single iteration of the algorithm. Repeated iterations 
are needed to reach the final answer but empirically the 
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number of iterations appears to grow only slowly with 
graph size, so overall running time is roughly linear in n 
for given k. 

The developments described here leave some questions 
unanswered. We have, for example, treated only the case 
of equal group sizes and not addressed how the algorithm 
might be generalized to the unequal case, although this 
does not appear to present any great challenges — unequal 
sizes merely add a constant to the positions of points in 
the spectral embedding, but the rest of the calculation 
follows through much as before. Another variant of the 
problem arises when the group sizes are not exactly equal 
but are allowed to vary within limits. This type of prob- 
lem could be addressed using the algorithm described in 
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